fix: rotate synthesized iOS taps into native screen space#804
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR adjusts synthesized tap/drag coordinates to account for interface orientation by rotating “interface-oriented” points into the device-native (portrait) coordinate space used by synthesized events.
Changes:
- Add
nativeSynthesizedPoint(...)to rotate points based onUIInterfaceOrientationvalues. - Update synthesized tap and swipe to use rotated coordinates.
- Expose an Objective-C helper to read the app’s
interfaceOrientation, and add a unit test for the rotation logic.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift | Adds point-rotation helper + updates synthesized gestures to use it; adds a rotation test. |
| ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.m | Adds interfaceOrientationForApplication: via dynamic selector lookup. |
| ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.h | Declares orientation helper and documents return values. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| switch interfaceOrientation { | ||
| case 3: // landscapeRight | ||
| return CGPoint(x: height - localY, y: localX) | ||
| case 4: // landscapeLeft | ||
| return CGPoint(x: localY, y: width - localX) | ||
| case 2: // portraitUpsideDown | ||
| return CGPoint(x: width - localX, y: height - localY) | ||
| default: // 1 portrait, 0 unknown | ||
| return CGPoint(x: localX, y: localY) | ||
| } |
| let localX = x - Double(frame.minX) | ||
| let localY = y - Double(frame.minY) |
|
Pushed a follow-up commit that applies the same native-coordinate conversion to synthesized transform gestures, not just tap/drag. This covers the pinch, rotate-gesture, and transform-gesture paths that all go through the private XCTest synthesis API. Also made the test app Gesture Lab rotation-capable and surfaced its gesture status in the visible landscape area so this can be verified against our own fixture. Validation I ran locally:
|
Summary
Synthesized iOS taps/drags land in the wrong place once the device is rotated — in landscape, taps on a sidebar row, keyboard key, or any element-by-ref miss their target.
XCUICoordinatetaps are unaffected.XCSynthesizedEventRecordpointer paths are consumed in device-native (portrait) screen coordinates, butsynthesizedTapAt/synthesizedDragAtpass the element frame's interface-oriented coordinates through unchanged (they coincide in portrait but differ by a 90° rotation in landscape). The synthesized path already passes the app'sinterfaceOrientationto the event record, but the path points still need to be in native space. From bisecting it looks like this bug was introduced by #702 and the fix should be in the scope of ADR 0005. The pre-#702XCUICoordinate.tap()was fine because XCTest rotates internally.Fix: rotate the point into native space using the app's
interfaceOrientationbefore synthesizing (identity fallback when the orientation can't be read, so a missing reading is never worse than today). AddstestNativeSynthesizedPointRotatesByInterfaceOrientationas a regression test.Test plan
See
testNativeSynthesizedPointRotatesByInterfaceOrientation.Deterministic unit test (no device) — passes with the fix, fails against the pre-fix pass-through.
To reproduce the original bug and its fix end-to-end on a simulator you can drive a rotation-supported app with a click target through:
Integration tests would require adding a rotation aware device test -- I have one locally to prove this out but maybe a bit heavy for this one case. Happy to add as a follow-up if useful.